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ABSTRACT 

A study was undertaken to assess the relationship 
between the DMI Mathematics Systems Objective Inventory (DMIHS) and 
the National Assessment of Educational Progress (NAEP) content by 
process matrix prepared for the Fourth Assessment of Mathematics* The 
DMIHS is a criterion**referenced standardized test conqprised of seven 
levels^ and it can be used to assess student performance from the 
middle of kindergarten to the end of ninth grade* Two mathematics 
educators participated in the stuoy and completed the same task* 
Findings indicate that the DNIMS reached from 17% to 48% of the 
NAEP's matrix cellSr figures that did not differ dramatically from 
earlier results on norm-referenced tests and an earlier NAEP system* 
The NAEP does not address each contents-process combination at each 
levels and comparing these figures to a 100% base would not be 
appropriate* These findings show that individuals concerned with 
student assessment in mathematics along the NAEP system will find 
that the DMIMS attends to the same interests as do the 
florm**referenced standardized tests on mathematics* Findings also have 
inqplications for inclusion in tests of items other than those that 
deal with operations with numbers* Thirteen tables are included^ and 
notes and references are appended* (TJH) 
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A COKTEKT ANALYSIS OF THE DMI MATHEMATICS SYSTEMS INSTRUCTIONAL 
OBJECTIVES IN^rEKTORY ACCORDING TO THE 1985-86 NATIONAL 
ASSESSMENT OF EDUCATIONAL PROGRESS CONTENT BY 
PROCESS MATRIX FOR MATHEMATICS 

The National Assessment of Educational Progress <NAEP) has conducted 
four mathematics assessments, 1972-73, 1978*79, 1981-82 and 1985-86.1 in 
the 1985-86 Assessment* the NAEP prepared a content by process matrix which 
covered the mathematics domain*^ Seven content and five process categories 
made up the matrix which was used to help prepare and classify the 
exercises constructed by the Assessment. In the same sense, the matrix can 
be used to study and classify the items included in the norm-referenced and 
criterion-referenced standardized mathematics tests marketed today. 
Earlier studies used the 1972-73 NAEP system,^ which was based on content 
alone, to classify the items used in norm-referenced standardized 
mathematics achievement tests designea to measure elementary^ and secondary 
school student achievement.^ This study will examine the content of a 
current criterion-referenctrd test seriej?^ the bmt Mat hematics Systems 
Tn.^t-.riif^t-.innai n>^jf^rtWP5^ Inventory (dmimS) , ^ according to the NAEP content 
by process matrix in order to determine the extent to which this test 
series follows the NAEP system. 

CpiVerion*Pef<>rf^nr;g»d Tfrshinc^ 

Criterion-referenced tests measure specific objectives which are 
stated as behavioral changes antici^^ated in the test^taker as a result of 
instruction^ while norm-referenced testing deals with relative status in 
the subject assessed. Most educators acknowledge the need for both forms 
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of assessment but differ with regard to the en5)hasis assigned to each. 

Advocates of criterion-referenced testing claim that normative 
achievement comparisons can be misinterpreted as aptitude comparisons,^ 
Additionally, the results of norm- referenced tests can be misconstrued; 
indicating fixed rather than malleable abilities. On the other hand, 
criterion-referenced tests focus on clearly defined behavioral samples and 
the results are not as susceptible to misinterpretation. The same items 
may be used to build norm- referenced and criterion-referenced tests, but 
criterion- referenced test items tend to be more detailed as the test 
constructor can work with a limited number of objectives. In the 
norm** referenced system, test constructors may try to examine several skills 
with a single item. 

Criterion- referenced testing has attracted researchers' attention over 
time. Five references appeared under Criterion Referenced Tests in the 
19d6-d7 edition of Eduriation Tndex ^ and eleven references appeared under 
the same title in Resources in EdunAhinn-lQ Despite the attention given to 
this topic, Mitchell cited only one commercially-prepared, 
criterion-*referenced mathematics achievement test series in the tJinth 
MAni>;^1 M^aftnrf>m^nr,ff Y^^rbook^ll ^he DMIMS . 

Criterion-referenced tests have become integral parts of school 
district assessment programs across the country. Rose and Custin described 
the development of a criterion-referenced testing program for the 
Charleston, South Carolina schools. 12 ^he school district prepared reading 
comprehension and general mathematics tests for students enrolled in grades 
one through eight, Jost et al* discussed the development and presented the 
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results of criterion-referenced and norm-referenced tests in Los Angeles*^^ 
The Denver schools used criterion-referenced tests to supplement a 
norm- referenced testing program which did not address all of the school 
district's instructional objectives. 1^ Glickman and Pajak studied a group 
of Georgia schools which showed iinprovement in criterion-referenced test 
scores over time.^^ These studies underscore the role of 
criterion-referenced testing in America's schools and show that this 
assessment strategy has earned a place in the nation's educational system. 

The DMT Mat'.h4:^mat'.ir*% SyRi:*>m 

According to CTB/McGraw-Hill, their experience with the PrescripLive 
Mathematics Tnv^ntrtryi^ ^j^^ the Diagnostif; Mafhf^tnatics Inventory (DMI) led 
to the development of the DMIMS*^® This test series provides information 
which helps educators place students for measurement, diagnose their 
instructional needs, prescribe instructional activities, teach skills, 
monitor progress and suggest reinforcement and enrichment exercises. 

The DMIMS is made up of seven levels and can be used to asses3 student 
performance from the middle of kindergarten to the end of ninth grade. 1^ 
Table 1 shows grade coverage and the number of items Contained in 
each level * 

The information shown in Table 1 is not unusual- Seven of the eight 
levels cover a year's time, from the middle of one g-jade to the middle of 
the next, and the highest level. Level G, covers two and one-half years or 
more, depending on its application* The item count increases from one 
level to the next, another feature followed by many test publishers. 
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Table 1 

The DMT Matihematica fiyj^hfimft Tn^tirurtiionaJ nK-tanhlvfts 
Inventory- Grade Coverage and Number 
of Items by Test Level 



Level 


Grade Coverage 


Number of items 


A 


K.6 - 1.5 


48 


B 


1.6 - 2.5 


64 


C 


2.6 - 3.5 


72 


D 


3.6 - 4.5 


88 


E 


4.6 - 5.5 


116 


F 


5.6 - 6.5 


132 


G 


6.6 - 8.9+ 


168 



Durost critiqued the DMIMS's predecessor, the DMI, and noted that the 
publisher did not provide nonns, a policy which could encourage school 
districts to construct their own*20 He saw iinplications for individualized 
instruction through the DMI but added that this ideal is rarely 
achieved^The critic believed that the instrument was at least sound as 
any other on the market at the time, but the lack of an item difficulty 
index was a serious problem. Durost said the items included in the DMI 
were excellent but the instrument was more an inventory than a test* 

O'Brien said that the DMI dealt with topics addressed in the textbooks 
which were available at the tiire.^^ These topics, however, focused on 
computational skills and neglected the topics which contemporary 
mathematicians would look on as important: 
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A major problem is that the 325 objectives upon which the DMI is 
based seem to be based on a view that 'mathematics knowing* is a 
static business based on memori' of procedures, nomenclature, and 
association rather than a dynamic affair involving the construction 
testing and generalizing of relationships. 

Hambleton and £lgnor called the DMI a popular test when they discussed 
guidelines for evaluating criterion referenced tests and their manuals /^^ 
Interestingly, the writers said that many criterion-referenced tests fell 
short of the technical quality neceddary for them to accomplish their goal. 
Or, the supporting measurement theory had not reached the level of 
sophistication demonstrated by c rite rion-^ referenced tests* Finally, 
Hambleton and Eignor mentioned the lack of guidelines for 
criterion-referenced test evaluation and prepared a list of 
desirable characteristics* 

According to Hanna, the DMI offers its users a systematic ireans for 
placing students in mathematics programs and levels, diagnosing their 
instructional needs, and prescribing proper instructional materials The 
information produced by the test results may be used to teach specific 
skills, monitor progress and reinforce and enrich those skills which have 
been mastered* Hanna liked the locator tests which are part of the DMI 
because their use can maximize utility in terms of testing time* "Locator 
tests match students with levels capable of p^joviding the most 
reliable scores* "25 

Alvarez used the DMI in a study designed to help identify second grade 
students at risk. 26 35^© also a^Jministered the Pr^aeript-^TTP Reading 
Inventory . Eight hundred twenty-two third grade students participated jn 
the study and Alvarez used multiple regression and stepwise discriminant 



anlysift to analyze her data. The researcher found that f rst grade test 
scores and report card grades were significant contributors to second grade 
test performance and achievement. Gender was not a factor. Thus, the DMI 
provided data which could be used for prediction. The instrument was 
effective and efficient. 

Constantine studied the influence of the number of test items on 
mastery decisions. He used the DMI to collect data^ Generally, between 
ten and twenty items ought to be used to measure objective mastery. 28 
Constantine prepared simulated test forms for the study. The r.umber of 
items used to mesure each objective ranged from zero or chance to tw>nty + 
One item represented DMI specific objectives, two to eight, DMI category 
objectives, four, revised DMI objectives, ten, Popham*s minimum, and 
twenty, his maximum. The researcher found that single item objectives 
generated significantly more consistent decisions than those attributed to 
chance. This finding occurred at all grade levels. Moreover, there were 
no significant differences between decisions made on the basis of ten or 
twenty items. Decisions based on more than one item were more consistent 
than those based on a single item. 

Alford commented on the relationship between textbooks and 
standardized test content and showed a need for the NAEP. Working through 
a pair of studies. Freeman et al. and Mehrens and Phillips, Alford found 
that the tests' contents varied. Since all of the tests revj.ewed 
emphasized computational skills and some of the textbook material was not 
attended to by the tests, the textbook"test match was weak. Therefore, 
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Alford continued^ teachi^rs must be aware o£ the discrepancy between textd 
and tests when they develop their instructional programs. If they neglect 
this relationship^ teachers who use their state guidelines as a frame o£ 
reference for their curriculum will deliver a different instructional 
program than those who use standardized tests or textbooks to fill this 
role. Those who use their state guidelines as an explicit curriculum will 
teach a different course than those who use standardized test or textbook 
content as an in^licit one. 

Alford suppported the use of standardized tests whil<» noting that 
their items only sample the appr<^riate domain. This donain is large and 
the tests seem to be representative. The textbooks used in mathematics 
teaching cover more material than that which is taught. "Thus, the picture 
emerging from studies of textbook and test content almost certainly is an 
overestimation of the match between the curricuJ.um students are taught and 
the tests they take."^o 

Alford* s comments are pertinent to the issue at hand because they 
reflect the confusion surrounding mathematics education. Perhaps the NAEP 
can help focus mathematics curricula in the schools and promote more 
uniform mathematics instruction across the country. In^roving the 
correlation between the subject matter taught and that tested will benefit 
teachers and their student!*}. 

The NAEP atten^ted to improve this relationship by constructing the 
content by process matrix which was designed to cover the mathematics 
domain. Each Assessment's objectives have reflected the positions of 
educators, mathematicians, researchers and the public. Moreover, each 
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Assessment has developed a character of its own as the information produced 
has been used \o focus on policy decisions, student competencies and the 
interrelationships among subjects. 

For the 1985-S6 Assessment, a group of tewnty-five mathematics 
educators and classroom teachers exeutiined the objectives set for the 
1961-62 Assessment. NAEP staff personnel summarized the results and 
submitted them to the Mathematics Learning Area Committee <MLAC) which 
updated the 1965-66 Assessment objectives, 31 The MLAC used the NAEP 
Assessment Policy Comntittee's guidelines in order to prepare the 
objectives. Here, the Policy Comntittee asked those responsible for setting 
up all the 1965-66 Assessments to emphasize the "higher-level* 
critical-thinking skills. "32 The MLAC draft was reviewed by another 
twenty-five member panel made up of mathematics educators and classroom 
teachers. The MLAC reviewed this panel's comments and produced the 
Assessment's objectives. 

Thft Objectives : Five objectives were defined for the Asessment: 
(1) Problem Solving/Reasoning* (2) Routine Application, <3) Understanding/ 
Comprehension, (4) Skill and (5) Knowledge. Some degree of problem solving 
was incorporated into each objective in response to the Policy Committee's 
interest in examining critical thinking skills. Elementary and secondary 
school mathematics up to but not including the calculus served as the 
Assessments content domain. The MLAC defined seven process categories: 
(1) Fundamental Methods of Mathematics, (2) Discrete Mathematics, (3) Data 
Organization and Interpretation* <4) Measurement* <5) Geometry, 
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(6) Relations, Functions, and Algebraic Expressions, and Numbers 
and Operations , 

Problem Solving/Reasoning was designed to measure a student's higher 
order thinking skills. Consequently, the exercises prepared to measure 
this skill were jjomewhat more intellectually complex than the ones used to 
measure skill application or basic understanding. Routine Application 
measured a student^s ability to solve familiar problems, those studied in 
class or textbooks and Understanding /Comprehension, asked students to 
demonstrate their facility in interpreting and elaborating on basic 
concepts, assumptions and relationships. Skill encompassed routine 
manipulations and Knowledge, recall and recognition. 

Mathematical Methods examined the processes crucial to mathematics. 
Deductive and inductive proof, logic and standard problem-^solving 
strategies were included in this objective. Discrete Mathematics focused 
on probability and matrix operations while Data Organization and 
Interpretation included central tendency measures and variance. 
Measurement took in mass, weight, area, volume, money and scale drawings 
and Geometry, the characteristics of geometric figures and formal proofs. 
Relations, Functions and Algebraic Expressions, a broad area, covered the 
use of variables in expressions, words and syml>ols and exponential and 
trigonometric functions. Numeration and number concepts were addressed by 
Numbers and Operations which offered problems in whole numbers, common and 
decimal fractions, integers and percent. 
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The researcher asked a CTB/McGraw-^Hill representative for a set of 
the DMIMS levels and its supporting materials* This request was answered 
and each item in Level D, a random choicei was examined and assigned to a 
cell in the HAEP content by process matrix. Two mathematics educators 
completed the same task. Agre^ent reached 95 percent as the classifiers 
made the same assignment for eighty-four of eighty-eight items. 
This figure established the classification procedure's reliability and the 
researcher completed this activity lor the r^smaining DMIMS levels. The 
HAEP, CTB/McGraw Hill and the researcher's experience established validity. 
Tables 2 through 8 present the results. 

Table 2 shows that Level A addressed seven of the thirty*-five cells in 
the ;Tiatrix: (1) Measurement*- Skill, (2) Measurements-Knowledge, 
(3) Geonietry-Routine Application^ (4) Geometry-Knowledge i 
(5) Humber-Routine ^plication, (6) Nuinber*Skill and (7) Humbert-Knowledge. 
For ProcesSf Skill was addressed by twenty*four of the forty-eight items 
(50 %) and for Content, Humber by twenty-four items (50 %) + 



ERIC 



12 



11 



Table 2 



A Classification of the Level A DMIMS Items according to the 
HAEP Content by Process Matrix 



Measurement 



Content 

Geometry Numbers & 
Operf tions 



Process 



Routine 
Applicat ion 

Skill 
Knowledge 



8 (17 %) 
8 (17 %) 
16 (33 %) 



4(8%) 

4(8%) 
8 {17 %) 



4(8%) 

16 (33 %) 

4(8%) 

24 (50 %) 



8 (17 %) 

24 (50 %) 

16 (33 %) 
48 

(100 %) 



Table 3 shows that Level B addressed six of the thirty^five cells in the 
matrix: {D Humber^Problem Solving^ (2) Measurement-Routine Applicat ion^ (3) 
number -Routine Application, (4) Number-Understanding* (5) Measurement-Skills 
and (6) Number-Skills. For Process* Skill accounted for thirty-six items 
(57 %) and for Content* Himber by forty-four items (69 %) . 
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Table 3 



A Classification of the Level B DMIMS Items according to the 
HAEP Content by Process Matrix 



Content 



Measurement 



NundDers & 
operations 



Process 



Problem 
Solving 

Routine 
Application 

Understanding 

Knowledge 



8 (12 %) 

12 (19 %) 
20 (31 %) 



4(6%) 

12 (19 %) 
4(6%) 

24 (38 %) 

44 (69 %) 



4(6% 

20 (31 % 
4(6% 
36 (S6 % 
64 

(100 % 



Table 4 shows that Level C attended to nine of the thirty-five matrix 
cells: (1) number-Problem Solving, (2) Data-Routine Application, 
(3) Measurement^Koutine Application, (4) Nuniber-Routine Application, 
(5) Measurement -Understanding, (6) Number-Understanding, (7) Geometry-Skill, 
(8) Humber-Skill and (9) Geometry- Knowledge* For Process, Skill accounted for 
thirty-nine items (54 %) and for Content, number, by fifty-four items (75 %) . 
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Table 4 

A Classification rf the Level C DMIMS Items according to the 
NA- Content by Process Matrix 

Content 

Data Org. Nuihbers & 

& Interp. Measurement Geometry Operations 

Process 

Problem 
Solving 

Rout ine 

Application 4 < 6 %) 

Understanding/ 
Comprehension 

Skill 
Knowledge 

4(6%) 



3(4%) 
3(4%) 

6(6%) 



4(6%) 
4(6%) 
e (11 %) 



5(7%) 

8 (11 %) 

6(8%) 

35 (49 %) 

5^ (73 %) 



5(7%) 

15 (21 %) 

9 (12 %) 

39 (54 %) 

4(6%) 

72 
(100 %) 



Table 5 shows that Level D addressed twelve of the thirty five cells in 
the matrix: (1) M-^chematical Met hods -Problem Solving, (2) Measurement "Problem 
Solving, (3) Number-Problem Solving, (4) Data-Routi.ie Application, 
(5) Nuiriber- Rout ine Application, (fi) Mathematical Methods-Understanding, 
(7) Data-Understanding, (6) Data-S)cill, (9) Measurement-SJcill, 
(10) Geometry-SJcill, (11) Number-SJcill and (12) Mathematical-Method. For 
Process, SJcill accounted for forty-eight items (54 %) ana for Content, Number, 
fifty-three items (60 %) . 
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Table 5 



A Classification of the Level D DMIMS items according to the 
NAEP Content by Process Matrix 



Mathematical 
Methods 



Content 



Data Org. 
& Interp, 



Measurement Geometry 



Process 

Problem 
Solving 

Routine 
Application 

Understanding/ 
Coitqprehension 

Skill 
Knowledge 



5 { 6 %) 



9 (10 %) 

1(1%) 
15 (1^ ^) 



2 ; 2 %) 

2(2%) 
4(4%) 

S ( 9 %) 



3(3%) 



5(6%) 4(4%) 



8(9%) 4(4%) 
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Table 5 (cont') 



Content 



HundDers & 
operations 



Process 
Problem 

Solving 7(8%) 15 (17 %) 

Routine 

Application 11 (12 %) 13 (15 %) 

Ui arstanding/ 

Coirprehension 11 (12 %) 

Skill 35 (40 %) 48 (54 %) 

Knowledge 1(1%) 



13 (60 %) 08 

(100 %) 



Table 6 shows that Level E addressed fifteen of the thirty*five 
Process-Content combinations: (1! Mathematical Method-Problem Solving, 
(2) Measurement-Problem Solving, (3) Number-Problem Solving, (4) Data-Routine 
Application, (5) Measurement -Routine Applicationr (6) Gecanetry-Routine 
Application, (7) Relat'ons-Routine Application, (8) Number-Routine 
Application, (9) Mathematical Method-Understanding, 

(10) Measurement -Understanding, (11) Measurement-Skiil, (12) Geometry-Skill/ 
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J 13) Number-Skill, (14) Measurement -Knowledge and <15) Geometry-Knowledge, For 
Process, Skill accounted for forty items <34 %) and for Content, Number, 
fifty-six items <48 %) , 

Table 6 

A Classification of the Level E DMIMS Items according to the 
NAEP Content by Procef^s Matrix 

Content 

Mathematical Data Org. 

Methods Interp. Measurement Geometry 



Process 
Problem 



Solving 


3(3%) 




7 


( 6 


%) 










Routine 
Application 




8(7%) 


2 


( 2 


%) 


2 


( 


2 


%) 


Understanding/ 
Con5>rehen3ioij 


16 (14 %) 




2 


( 2 


%) 










Skill 






4 


( 3 


%) 


4 


( 


3 


%) 


Knowledge 






6 


( 5 


%) 


2 


( 


2 


%) 




19 (16 %) 


8 (7 %) 


21 


(18 


%) 


8 


( 


7 


%) 



18 



17 



Table 6 (cont,) 



Relat ions 
Functions + 



Content 

Numbers £ 
Operations 



Process 

Problem 
Solving 

Rout ine 
Application 

Understanding/ 
Con^rehension 

Skill 

Knowledge 



4(3%) 



4(3%) 



12 (10 %) 
12 (10 %) 



32 (28 %) 



56 (48 %) 



22 (19 %) 

28 (24 %) 

18 (16 %) 
40 (34 %) 
8(7%) 
116 

(100 %) 



Table 7 shows that Level F attended to thirteen of the thirty-five 
Process -Content combinations: (1) Mathematical Method-Problem Solving, 
(2) Data-Problem Solving, (3) Huitber-Problem Solving, (4) Mathematical 
Method-Routine Application, (5) Data^Ro^itine Application, 
(6) Measurement-Routine Application, (7) Geometry-Routine Application, 
(8) Relations-Routine Application, <9) Mathematical Method-Understanding, 
(10) Measurement-Skill, (11) Geometry-Skill, (12) Humber-Skill and 
(13) Geometry-Knowledge, For Process, Skill accounted for forty-seven items 
(36 %) and for Content, Humoer, sixty-five items (50 %) - 
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Table 7 



A Classification of the Level F DMIMS Items according to the 
NAEP Content by Process Matrix 



Mathematical 
Methods 



Content 

Data Org< 
& Interp< 



Measurement Geometry 



Process 

Problem 
Solving 

Rout ine 
Application 

Unde rst anding/ 
Comprehension 

Skill 

Knowledge 



6(6%) 
4(3%) 
14 (11 %) 



2 <2 %) 



3 <2 %) 



4(3%) 12 { 9 %) 



26 (20 %) 



4(3%) 6(5%) 
5(4%) 

5(4%) 8(6%) 23 (18 %) 
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Table 7 (cont . ) 



Content 

Relations Numbers & 

Functions + Operations 



Process 
Problem 

Solving 28 (73 %) 38 (29 %J 

Routine 

implication 4(3%) 27 (21 %) 

Understanding/ 

Comprehension 14 (11 %) 

Skill 38 (28 %) 47 (36 %) 

Knowledge 5(4%) 

4(3%) 66 (50 %) 132 

(100 %) 



Table 8 shows that Level G responded to seventeen of the thirty-five 
Content by Process matrix cells: (1) Mathematical Method-Problem Solving, 
(2) Data-Problem Solving, (3) Geometry-Problem Solving, (4) Relations -Problem 
Solving, (5) Number-Problem-SolviDj, (6) Discrete Mathematics-Routine 
Application, (7) Data-Routine Application, (8) Measurement-Routine 
Application, (9) Geometry-Routine Application, (10) Relations -Routine 
Application, (11) Number-Routine Application, (12) Mathematical 
Method-Understanding, (13) Geometry-Skill, (14) Relations-Skill, 
(15) Number-Skill, (16) Measurement -Knowledge and (17) Geometry-Knowledge. 
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For Process, Problem Solving accounted for fi£ty*one items (30 %) and for 
Content, Numbers, eighty^four items (50 %) . 



Table 6 



A Classification of the Level G DKIMS items according to the 
NAEP Content by Process Matrix 



Mathematical 
Met hods 



Content 

Discrete 
Mathematics 



Data Org. 
& interp. 



Measurement 



Process 

Problem 
Solving 

Routine 
Application 

Understandi ng/ 
Cofnpre hens ion 

Skill 

Knowledge 



8(5%) 



20 (12 %) 



28 (17 %) 



4(2%) 



4(2%) 
2(1%) 



5(3%) 



4(2%) 



6(4%) 



3(2%) 
8(5%) 
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Table 8 (cont J 



Content 

Relations Numbers £ 
Geometry Functions + Operations 



Process 
Problem 

Solving 1(1%) 1(1%) 37 (22 %) 51 (30 %) 

Routine 

Application ^0(6%) 12 ( 7 %) 12 ( 7 %) 45 (27 %) 

Unde r s t a ndi ng / 

Coinprehension 20 (12 %) 

Skill 8(5%) 4(2%) 35 (21 %) 47 (28 %) 

Knowledge 2(1%) 5(3%) 

21 (12 %) 17 (10 %) 84 (50 %) 168 

(100 %) 



The number of matrix cells addressed by the DMIMS seven levels ranged 
from six in Level B (17 %) to seventeen (48 %) in Level G. The mean for the 
levels was eleven (31 %) . These percents approximate those found by the 
researcher in his study on norm*referenced tests at the elementary school 
level. Here, the objectives prepared by the NAEP for its first mathematics 
assessment served as the criterion. There were seventeen content objectives 
and the number addressed by each test ranged from three (17 %) to 
nine (52 %) .^^ 



22 



ERIC 



Since the NAEP assessed the mathematics performance of nine-year old anc 
thirteen-year old students, the researcher analyzed the DMIMS levels which 
would be used with students at these ages. For nine-year olds who would 
probably be enrolled in fourth grade* Levels D and E would be appropriate. 
For thirteen-year olds. Level G would be the DMIMS level used. Tables 9 and 
10 show these findings expressed as percents of Content and Process Items for 
Levels D and E and the NAEP.^* Tables 11 and 12 take the same approach for 
Level G and the NAEP . 



4 



Table 9 



Distribution of Level 6 DMIHS Items and NAEP Content 
Objective Items for Nine-Year Old Students 
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Level D 



Test 
Level E 



NAEP 



Objective 



Mathematical 
Methods 

Discete 
Mathematics 

Data Organization 
& Interpretation 

Measurement 

Geometry 

Relations^ 
Functions + 

Nuinbers & 
Operations 



17 I 



9 I 
9 % 
4 I 



16 % 



60 % 



7 % 
18 I 
7 I 

3 I 

48 % 



12 % 

2 % 

11 I 
19 % 
5 I 

5 I 

47 I 
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Table 10 



Distribution of Level G DMIMS Items and NAEP Process 
Objective Items for Nine-Year Old Students 
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Objective 

problem 
Solving 

Routine 
Application 

Understanding/ 
Comprehension 

Skill 

Knowledge 



Level D 



17 % 

15 % 

12 % 
54 % 
1 % 



Test 
Level E 



19 % 

24 % 

16 % 
34 % 
7 % 



NAEP 



17 % 

12 % 

19 % 
40 % 
12 % 
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Table 11 



Distribution of Level G DMIMS Items and NAEP Content 
Objective items for Thirteen-Year Old Students 



Test 

T^vel G NAEP 



Objective 



Mathematical 

Methods 17 % 1^ % 

Discrete 

Mathematics 2 % 4 % 

Data Organization 

& Interpretation 4 % 8 % 

Measurement 5 % 17 % 

Geometry 12 % 8 % 

Relations/ 

Functions + 10 % 5 % 

Numbers & 

Operations SO % 49 % 
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Table 12 



Distribution of Level G DMIMS Items and NAJP Process 
Objective Items for Thirteen-Year Old Studer/.e 



Objective 



Test 

Level G NAEP 



Problem 

Solving 30 % 18 % 

Routine 

Application 11 % 15 % 

Understanding/ 

Comprehension 12 % 25 % 

Skill 47 % 32 % 

Knowledge 3 % 11 % 



The data presented in Tables 9 through 12 were analyzed by Spearman^s 
ranlc-ord<^r correlation procedure and the results appear in Table 13. These 
data show that the correlations for Level T>f Level E and the Assessment 
exceeded .90 in Content. For Level G and the Assessment^ the figure was .67. 
For Process^ the correlations ranged from .46 to .70. The correlations for 
Content were nigher than those for Process. Level D had a stronger 
relationship with the Assessment than Level £ for nine-year old students. 
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Table 13 



Spearman Rank-Order Correlation Coefficients: DMIMS Levels D, E and 
NAEP Nine-Year Cld Item Correlations and DMIMS Level G 
and NAEP Thirteen Year-Old Item Correlations 



Instruments Content Process 



Level D - Assessment 
Nine-Year Olds 
Level E - Assessment 
Nine Year^Olds 
Level G - Assessment 
Thirteen Year-Olds 



,91 .61 
,97 -46 
,67 ,70 



CQncluisions: 

This rjtudy examined the relationship between the DM I Mathematics Systems 
Objective Inventory (DMIMS) r a criterion-referenced standardized test, and the 
National Assessment of Educational Progress Content by Process (NAEP) matrix 
prepared fo^^ the Fourth Assessment of Mathematics, The findings showed that 
the DMIMS reached from seventeen percent to forty-eight percent of the NAEP*s 
matrix cells, figures which did not differ dramatically from the r£^ssarcher*s 
efforts with norm- referenced tests and an earlier NAEP system. The NAEP does 
not address each Content^Process combination at each level and comparing these 
figures to a 100 p cent base would not be appropriate. These findings show 
that individuals concerned with student assessment in mathematics along the 
NAEP system will find that the DMIMS attends to the same interests as the 
norm- referenced standardized tests on mathematics. 
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The investigator's efforts with norm'-referenced standardized tests are 
dated and may not serve as an appropriate comparison. Perhaps the current 
norm-referenced tests are more in line with the NAEP system in mathematics* 
This matter can be resolved by further study within the framework used in this 
investigation* 

Mathematics educators have called for courses of study which emphasize 
interests in mathematics o^her than operations with numbers. Unfortunately/ 
the mathematics textbooks and standardized tests continue to emphasize 
operations with numbers* While other areas are addressed, operations with 
numbers account fo^ roughly half the items in standardized tests* 

Numbers and Operations is one of the NAEP's content categories* The 
percentage of DMIMS items assigned to this category ranged from 48 percent in 
Level E to 75 percent in Leve) C* The Assessment's Numbers and Operations 
category for nine-year olds included 47 percent of its exercises and for 
thiLteen-year oldSf 49 percent* While it is difficult to dispute the need for 
facility with numbers in today's society/ the apparent discrepancy between the 
constructors of the NAEP's exercises who emphasize operations with numbers and 
those who call for adjustments in this emphasis ought to be resolved. 
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